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Abstract. In this article we test the accuracy of three plat- 
forms used in computational modelling: MatLab, Octave and 
Scilab, running on i386 architecture and three operating sys- 
tems (Windows, Ubuntu and Mac OS). We submitted them to 
numerical tests using standard data sets and using the func- 
tions provided by each platform. A Monte Carlo study was 
conducted in some of the datasets in order to verify the sta- 
bility of the results with respect to small departures from the 
original input. We propose a set of operations which include 
the computation of matrix determinants and eigenvalues, whose 
results are known. We also used data provided by NIST (Na- 
tional Institute of Standards and Technology), a protocol which 
includes the computation of basic univariate statistics (mean, 
standard deviation and first-lag correlation), linear regression 
and extremes of probability distributions. The assessment was 
made comparing the results computed by the platforms with 
certified values, that is, known results, computing the number 
of correct significant digits. 
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1. Introduction 

Mathematical modelling aims at solving complex problems which 
can be described in a rigorous mathematical way that enables the 
use of computers for finding the solution. 

Many mathematical models which arise in diverse areas as engi- 
neering, bioinformatics and ecology rely on partial differential equa- 
tions (PDE) or ordinary differential equations (ODE), where the high 
number of variables requires strong computational effort in their so- 
lution. 

The final output is the result of a myriad of often disregarded 
intermediate computations. To illustrate this, when a finite element 
mesh is used to perform a structural analysis, computing matrix 
inversions and determinants are important commonplace operations 
which are rarely checked. 

The search for best approximate solutions, considering some rea- 
sonable bounds of errors, imposes tight accuracy requirements on the 
computational platforms and its libraries or functions. When deal- 
ing with huge structures described by irregular meshes, for instance, 
algorithms for domain partitioning and parallel computing are often 
needed, and the correctness of the results is still more critical. Such 
partitioning algorithms are usually based on either topological or 
spectral methods, which assess the algebraic properties of the graph 
associated to the mesh [U [16]. That is, the mesh can be associated 
to a dual graph, such that the vertices correspond to the finite ele- 
ments and the edges represent the connectivity of the elements which 
share common boards. If a graph is connected, then it is shown that 
the second eigenvalue of its Laplacian matrix is positive [7]. The 
components of the second eigenvector are associated with the corre- 
sponding vertices of the graph and can be used to assign weights for 
partitioning of the graph. 

In complex problems with many variables and values, minute er- 
rors in obtaining the eigenvalue and eigenvector or a matrix deter- 
minant, in calculating an average, a standard deviation or a corre- 
lation coefficient, can lead to erroneous decisions. Computational 
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platforms offer libraries and functions for carrying out these calcu- 
lations. When it comes to modelling large problems, with complex 
variables, good, or at least controlled, responses are fundamental. 

Little attention has been drawn to assess these platforms under 
the diversity of operational systems and hardware considering the 
accuracy of the results. Examples of such assessments are found in 
[3j HI EJ |9]. Most of these studies, usually limited to spreadsheets, 
follow the methodology suggested by McCullough [TO] lU\ H2l [i3j Hi] : 
constasting results with the certified values provided by the Statisti- 
cal Reference Datasets (StRD) produced by the National Institute of 
Standards and Technology (NIST) [15J. We add a Monte Carlo study 
to the protocol in some of the datasets in order to verify the stabil- 
ity of the results with respect to small departures from the original 
input. Besides statistical tests, in [8] we proposed additional tests 
that employ operations on matrices in order to assess the scientific 
platforms from this viewpoint. 

In this work we test three numerical scientific platforms: Oc- 
tave 3.2.4, Scilab 5.3 and Matlab R2011a, under the three well 
known operating systems: Windows XP Professional SP 2, Linux 
Ubuntu 10.4 and Mac OS X Leopard 10.5.6, whenever the former 
are available. In all cases, i386 architecture hardware was employed, 
and double precision computation was enforced. 

Outline. The remainder of this work is organized as follows. Sec- 
tion [2] discusses how accuracy is measured. Section [3] presents the 
results obtained assessing basic statistics (subsection 13.11) . proba- 
bility distribution functions (subection I3.2[) . linear regression (sub- 
section [3]3]) and operations on matrices (subsection 13 .4p . Section 0] 
concludes the paper. 

2. Measuring Accuracy 

Errors in computational simulations can occur and arise from 
diferents sources. They range from modelling errors, defined by the 
difference between real world and the computable model, to numer- 
ical errors introduced in the solution of the problem. The latter 



4 



E. S. ALMEIDA ET AL. 



are (i) round-off errors, (ii) truncation and discretization errors, or 
(iii) numerical instability. Usually the availability to implementation 
details of the algorithms is very limited and, even when available, 
other factors, like hardware, compiler, operational system, may com- 
promise the software accuracy. 

Considering such limitations, many authors (see, for instance, 
[3J HI El [9]) adopt the strategy of measuring the software accuracy 
from the user viewpoing, that is, comparing the results provided by 
the software with certified values known to be correct. The certified 
values and datasets employed in this study are obtained in the Sta- 
tistical Reference Datasets from the National Institute of Standards 
and Technology (NIST) [15]. We also measure the stability of the 
results by Monte Carlo, and propose a strategy that considers the 
results of operations on matrices. 

In the first strategy, statistical descriptive measures are assessed: 
the mean, the standard deviation and the first lag coefficient of au- 
tocorrelation. Linear regression and quantiles of tail probabilities of 
usual distributions are also computed. 

The LRE (base-10 logarithm of the absolute value of the relative 
error) is computed as a measure of the accuracy of the functions. 
LRE is approximately the number of matching significant digits be- 
tween the certified and obtained values: 



where x is the result of evaluation function computed by the software 
under assessment and c is the certified value. 

The following convention was adopted: when LRE(x, c) > 1 we 
consider only one decimal place. If < LRE(x, c) < 1, it is assumed 
zero, that is, no correct digit was found. If the value was very far 
from the certified, "-" is used; the word 'Inf is used to mean that 
there is a perfect match, and when the platform returns an error, it 
is denoted by 'NA'. 



(1) 




if c^O, 
otherwise, 
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LREs were computed using the R platform Qhttp : / /www . r-pro j ect . org/ ) , 



whose excellent numerical properties were checked in [3]. 

In order to assess the stability of the results, we propose a Monte 
Carlo procedure for some of the datasets used to compute the preci- 
sion of statistical descriptive measures. A bootstap estimate of the 
LRE produced by each platform for each measure (mean, standard 
deviation and first lag coefficient of autocorrelation) was computed 
for each of the four real-world datasets plus PiDigits. One hun- 
dred independent vectors of the same size of the original ones were 
obtained sampling with reposition from each original data set. Certi- 
fied values were computed using R. These vectors were submitted to 
each platform, and the resulting observed quantities were contrasted 
with the certified values producing 100 LREs for each quantity of 
interest. These last values were used to compute an estimate of the 
standard deviation: 



where LRE denotes the "true" logarithm of the absolute value of the 
relative error, which was observed with the original dataset. This 
procedure does not belong to the original protocol, but it allows 
verifying how stable the resuts are with respect to relatively small 
perturbations of the dataset. 

The second strategy deals with operations on matrices. We pro- 
pose two assessments: the first is to compute the determinant of a 
2x2 matrix whose certified value is zero \M\ = 0. Consider the 
matrix 



with arbitrary values 6, s, e. 

The numerical computation of the determinant with built-in func- 
tions will guarantee that the intermediate values (be) and (s/e) are 
evaluated. The values we proposed to be assessed are b = 1(P and 




100 
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8 = lQ-i, with j 6 0, 1, . . . , 15, and 

£ = 0.9_^_9, k E {1, . . . , 15}. 

k times 

The measure of accuracy is the result of the logical comparison of the 
computed value and zero in the platform under assessment. That 
is, the interest in such case is not the value itself but the result of 
comparison. 

We also propose another assessment considering spectral graph 
theory. The interest in this proposal is that the Laplacian matrix 
is directly related to many properties of the graph as, for instance, 
connectivity [5]. 

Let G = (V, E) be a non-directed finite graph without loops such 
that V = {wi, w 2 , • • • , w n } is the set of vertices and E is the set of 
edges. Denote by deg(wj) the degree of vertex u>j. Let D be the 
diagonal degree matrix with entries deg(wj) and A be the adjacency 
matrix with elements a^, which take value 1 if there is an edge 
between Wi and Wj. The Laplacian matrix C(G) of G is the difference 
between D and A, i.e., C{G) = D — A. Fiedler [7j noted that: 

• The number of zero eigenvalues of C(G) is the number of 
connected components in the graph; 

• If Ai, . . . , A n are the eigenvalues of C(G), then = Ai < A2 < 
• • • < A n . 

• If the second smallest eigenvalue A2 is greater then zero, A2 > 
0, then G is connected and A2 is called algebraic connectivity. 

Considering algebraic connectivity, we propose an accuracy as- 
sessment based on the class of complete bipartite graphs. In such 
graphs we have two subsets of vertices, say V% and V2, such that no 
connections exists between vertices belonging to the same subset, 
and each vertex from V\ is connected to every vertex from V2. Let 
m and n be the cardinality of V\ and V2, respectively. If we denote 
this bipartite graph by K mjTl , its Laplacian matrix has the following 
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form: 



/ n 




(2) C(K n>n ) 





n 








-1 \ 
-1 



n -1 
— 1 —1 m 

-1 -1 



\ -1 ■ ■ ■ -1-1 u ■ • • my 

For this Laplacian matrix, Bolloboas [5] showed that the eigenvalues 
are Ai = 0, A m+n = m + n and there are n — 1 eigenvalues whose 
value is m, and m — 1 eigenvalues whose value is n. 

In order to do the assessment, we considered two special cases: 
one with almost perfect balance K m>m+ \, and other with almost the 
worst possible balance Ki,im-\, where m G {9,99,999}. We formed 
examples of three sizes of graphs: small, medium and big. The 
assessment is based upon the observation of seven quantities: 

(i) the LRE of the smallest eigenvalue (Ai = 0) denoted £%, 

(ii) the LRE of the biggest eigenvalue (A m+n = m + n) denoted 

(iii) the LRE of the sum of the eigenvalues (Y2T +n ^* = 2mn) 
denoted £5, 

(iv) the minimum LRE of those eigenvalues that should take value 
n (there are m — 1 of them) denoted £ n , 

(v) the minimum LRE of those eigenvalues that should take value 
m (there are n — 1 of them) denoted £ m , and 

(vi,vii) the percentage of eigenvalues which test equal to m and to 
n (being the correct answers n — 1 and m — 1, respectively) 
denoted In and £m- 



3. Results 



In this section we present the results of applying the two strategies 
described in the previous section. The tables present the accuracy of 
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the three programming ambients under assessment running (when- 
ever available) under Windows ('Win'), Linux ('Lin') and Mac OS 
('Mac'), in 32 and 64 bits architecture. 

3.1. Basic Statistics. The univariate summary statistics we as- 
sessed are the sample mean, the sample standard deviation and the 
sample first lag correlation of nine datasets. These datasets are clas- 
sified by NIST in three levels of numerical difficulty: low, average 
and high. The datasets with low difficulty are Lew, Lottery, Mavro, 
Michelso (these four datasets come from real world experiments), 
NumAccl and PiDigits. The average difficulty datasets are NumAcc2 
and NumAcc3, while NumAcc4 is the only high difficulty dataset. The 
certified values were calculated using multiple precision arithmetic 
to obtain 500 digits answers. 

The command mean is common to all platforms. The standard 
deviation in Octave and MatLab is computed with the command 
std, whereas the Scilab command is st_deviation. For comput- 
ing the correlation, Scilab provides the function correl which, sur- 
prisingly and in spite of what is informed in the documentation, 
returns the covariance rather than the correlation; the correlation 
was obtained dividing this result by the product of the sample stan- 
dard deviations of the subvectors. In Octave we used the command 
Correl(v(l :n-l) , v(2:n)), and in Matlab the command applied 
was corr (v(l :n-l) , v(2:n)), considering in both the vector v of 
size n > 3. 

The values in parenthesis are the bootstrap estimates of the stan- 
dard deviation of LREs, slre- Whenever 'Inf ' was observed, LRE = 
16, i.e., the highest possible accuracy in double precision, was used. 

3.2. Statistical Functions. The distributions herein assessed are 
the binomial (Table EJ), Poisson (Tables |3] and HJ), gamma (Table EJ, 
normal (Table EJ, \ 2 (Table ED, beta (Tabled]), t-Student (Table E) 
and F (Table DE). 

The commands to compute all the distributions, except for the 
beta in Matlab and Octave, are the same: binocdf for the binomial, 
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Table 1 . LREs for the basic statistics and bootstrap 
estimates of selected standard deviations 
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Sample Mean 





Windows 


Inf(O) 


Inf(0.99) 


lnf(0) 


Inf(0.04) Inf(0.04) 


Inf 


Inf 


Inf 


Inf 


Octave 


Linux 


Inf(0.04) 


Inf(10.44) 


Inf(11.45) 


Inf(11.35) Inf(10.94) 


Inf 


Inf 


Inf 


Inf 




Mac Os 


Inf(O) 


Inf(0.99) 


lnf(0) 


Inf(0.33) Inf(0.43) 


Inf 


Inf 14.0 


Inf 


MatLab 


Windows 


16.0(0) 


15.1(0.89) 


16.0(0) 


16.0(0.35) 15.4(10.87) 


16.0 14.0 15.0 13.9 




Linux 


16.0(0) 


16.0(0.03) 


16.0(0) 


16.0(0.35) 16.0(11.48) 


16.0 14.0 14.0 13.9 




Windows 


Inf(0.04) 


8.1(6.28) 


lnf(0) 


Inf(0.04) Inf(0.04) 


Inf 


Inf 


Inf 


7.7 


Scilab 


Linux 


Inf(O) 


8.1(7.92) 


lnf(0) 


Inf(0.33) Inf(0.43) 


Inf 


Inf 


Inf 


7.7 




Mac OS 


lnf(0) 


8.1(7.92) 


lnf(0) 


Inf(0.33) Inf(0.43) 


Inf 


Inf 


Inf 


7.7 


Sample Standard Deviation 




Windows 


Inf(0.88) 


Inf(0.24) 


Inf(0.87) 


13.1(2.94) 13.9(1.96) 


Inf 


Inf 


9.5 


8.3 


Octave 


Linux 


Inf(11.08) 


Inf(9.88) 


Inf(10.80) 13.1(10.88) 13.9(10.88) 


Inf 


Inf 


8.3 


Inf 




Mac Os 


Inf(1.76) 


Inf(0.35) 


Inf(0.42) 


13.1(2.75) 13.8(1.99) 


Inf 


Inf 


9.5 


8.3 


MatLab 


Windows 


14.8(0.70) 


16.0(0.35) 


15.2(0.53) 


13.8(2.06) 13.9(12.85) 


16.0 16.0 


9.4 


8.2 




Linux 14.8((0.70) 


16.0(0.35) 


16.0(0.41) 


13.8(2.06) 13.9(12.85) 


16.0 16.0 


9.4 


8.2 




Windows 


7.9(4.79) 


8.1(6.12) 


8.2(6.01) 


4.1(6.33) 6.2(6.5) 


Inf 


Inf 


Inf 


Inf 


Scilab 


Linux 


7.9(8.01) 


8.1(7.67) 


8.2(7.5) 


4.1(11.79) 6.2(9.62) 


Inf 


Inf 


Inf 


Inf 




Mac OS 


7.9(6.47) 


8.1(7.67) 


8.2(7.41) 


4.1(11.59) 6.2(9.62) 


Inf 


Inf 


Inf 


Inf 


Sample First Lag Correlation 




Windows 


4.0(0.48) 


2.1(0.67) 


2.6(0.75) 


1.8(0.56) 3.6(1.90) 





3.0 


3.0 


3.0 


Octave 


Linux 


4.0(2.10) 


2.1(0.60) 


2.6(0.50) 


1.8(0.55) 3.6(1.81) 





3.3 


3.3 


3.3 




Mac Os 


4.0(1.05) 


2.1(0.67) 


2.6(0.75) 


1.8(0.58) 3.6(1.52) 





3.0 


3.0 


3.0 


MatLab 


Windows 


3.9(6.06) 


2.0(3.49) 


2.6(4.08) 


1.7(2.78) 3.5(4.82) 





3.3 


3.3 


3.3 




Linux 


3.9(6.06) 


2.0(3.49) 


2.6(4.08) 


1.7(2.78) 3.5(4.82) 





3.3 


3.3 


3.3 




Windows 








0(1.75) 0(2.09) 


0.5 











Scilab 


Linux 








0(2.09) 0(2.09) 


0.5 













Mac OS 




0(2.39) 


0(2.40) 


0(1.75) 0(2.09) 


0.5 












poisscdf for the Poisson, gamcdf for the gamma, norminv for the 
normal, cdfchi for the \ 2 > tinv for the t-Student, and finv for 
the F law. The command beta_cdf computes beta distribuition 
in Octave, while Matlab provides the command betainv. Scilab 
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provides cdfbin, cdfpoi, cdfgam, cdfnor, cdfchi, cdfbet, cdft 
and cdf f for computing the binomial, Poisson, gamma, normal, x 2 > 
beta, t-Student and F quantiles, respectively. 

Table 2. Binomial distribution, n = 1030 and p — 1/2 



k 


Pv(X < k) 
Certified 


Matlab 




Octave 






Scilab 




Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 


1 


8.96114E-308 


3.0 


3.0 











Inf 


lnf 


lnf 


2 


4.61499E-305 


3.0 


3.0 











8.0 


8.0 


8.0 


100 


1.39413E-169 


1.0 


1.0 











7.0 


7.0 


7.0 


300 


2.91621E-42 

















7.0 


7.0 


7.0 


400 


3.89735E-13 











1.0 


4.0 


6.0 


6.0 


6.0 


410 


3.19438E-11 











6.0 


6.0 


6.0 


6.0 


6.0 



Table 3. Poisson probabilities, A = 200 



k 


Pr(X = k) 
Certified 


Matlab 




Octave 






Scilab 




Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 





1.38390E-87 


5.6 


5.6 


6.0 


6.0 


6.0 


7.0 


7.0 


7.0 


103 


1.41720E-14 


1.4 


1.1 


1.0 


1.0 


1.0 


2.0 








315 


1.41948E-14 


























400 


5.58069E-36 


6.4 


6.4 


6.0 


6.0 


6.0 











900 


1.73230E-286 


6.0 


6.0 


6.0 


6.0 


6.0 












Table 4. Poisson distribution functions, A = 200 



k 


A 


Pr(X < fe) 
Certified 


Matlab 




Octave 






Scilab 




Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 


1E+05 


1E+05 


0.500841 


1.0 


1.0 


1.0 


1.2 


1.0 


7.0 


7.0 


7.0 


1E+07 


1E+07 


0.500084 


7.0 


7.0 


7.0 


7.0 


7.0 


6.0 


6.0 


6.0 


1E+09 


1E+09 


0.500008 


7.0 


7.0 


7.0 


7.0 


7.0 


6.0 


6.0 


6.0 



3.3. Linear Regression. NIST offers eleven datasets to perform 
linear regression analysis. The datasets are divided into numerical 
difficulty levels: two of low level, (Norris and Pontius), two of 
average level, (Nointl and Noint2) and seven of high level. Table [TT1 
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Table 5. Gamma distribution functions, f3 — 1 





Pr(X < x) 


Matlab 


Octave 






Scilab 




x a 


Certified 


Win 


Lin 


Win Lin 


Mac 


Win 


Lin 


Mac 


0.1 0.1 


0.827552 


7.0 


7.0 


7.0 7.0 


7.0 


7.0 


7.0 


7.0 


0.2 0.1 


0.879420 


6.0 


6.0 


6.0 6.0 


6.0 


6.0 


6.0 


6.0 


0.2 0.2 


0.764435 


6.0 


6.0 


6.0 6.0 


6.0 


6.0 


6.0 


6.0 


0.4 0.3 


0.776381 


6.0 


6.0 


6.0 6.0 


6.0 


6.0 


6.0 


6.0 


0.5 0.4 


0.748019 


6.0 


6.0 


6.0 6.0 


6.0 


6.0 


6.0 


6.0 


Table 6. Normal quantiles, \x -- 


= and cr=l 








Matlab 


Octave 






Scilab 




V 


Certified z p 


Win 


Lin 


Win Lin 


Mac 


Win 


Lin 


Mac 


5E-1 





Inf 


Inf 


Inf Inf 


Inf 











1E-198 


-30.0529 


7.0 


7.0 


7.0 7.0 


7.0 


3.0 


3.0 


3.0 


1E-300 


-37.0471 


7.0 


7.0 


7.0 7.0 


7.0 


1.0 


1.0 


1.0 



Table 7. The x 2 distribution 



V 


n 


Pr(X > x) = p 
Certified x 


Matlab 




Octave 






Scilab 




Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 


2E-1 


1 


1.64237 


5.6 


5.6 


5.6 


5.6 


5.6 


5.3 


5.3 


5.3 


1E-7 


1 


28.3740 


6.4 


6.4 


6.4 


6.4 


6.4 


6.4 


6.4 


6.4 


1E-7 


5 


40.8630 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 


1E-12 


1 


50.8441 


6.6 


6.6 


7.1 


7.1 


7.1 


6.3 


6.3 


6.3 


0.48 


778 


779.312 


6.3 


6.3 


4.4 


4.4 


4.4 


6.2 


6.2 


6.2 


0.52 


782 


779.353 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 


6.3 



presents the smallest LRE of each regression, and the LRE of the 
residual standard deviation (RSD) of each fit. 

Octave and Matlab do not provide explicit functions for perform- 
ing linear regression. Rather than that, linear regression is computed 
solving a least squares problem, and the data requires prior prepa- 
ration for that. Scilab provides the function reglin to obtain the 
coefficients and RSD. 
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Table 8. Beta quantiles, a = 5 and /3 — 2 



Matlab Octave Scilab 



V 


Certified 


Win 


Lin 


Win 


Lin Mac 


Win 


Lin 


Mac 


1E-2 


2.94314E-01 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-3 


1.81386E-01 


6.1 


6.1 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-4 


1.12969E-01 


5.4 


5.4 


5.0 


5.0 


5.0 


5.0 


5 


5 


1E-5 


7.07371E-02 


6.2 


6.2 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-6 


4.44270E-02 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-7 


2.79523E-02 


5.9 


5.9 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-8 


1.76057E-02 


6.3 


6.3 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-9 


1.10963E-02 


5.5 


5.5 


5.0 


5.0 


5.0 


5.0 


5.0 


5.0 


1E-10 


6.99645E-03 


6.7 


6.7 


7.0 


7.0 


7.0 


7.0 


7.0 


7.0 


1E-11 


4.41255E-03 


6.7 


6.7 


7.0 


7.0 


7.0 


7.0 


7.0 


7.0 


1E-12 


2.78337E-03 


5.9 


5.9 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-13 


1.75589E-03 


6.1 


6.1 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-100 


6.98827E-21 


6.8 


6.S 


7.0 








7.0 


7.0 


7.0 


Table 9. The t-Student distribution, n 


= 1 






Pr(X > x) = p 


Matlab 




Octave 






Scilab 




V 


Certified x 


Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 


1E-8 


3.18310E+07 

















6.0 


6.0 


6.0 


IE- 11 


3.18310E+10 

















6.0 


6.0 


6.0 


1E-12 


3.18310E+11 












6.0 


6.0 


6.0 


1E-13 


3.18310E+12 

















6.0 


6.0 


6.0 


1E-100 


3.18310E+99 












8.0 


8.0 





Table 10. 


The F distribution, ni = 


= n 2 ■ 


= 1 






Pr(X > x) = p 


Matlab 




Octave 






Scilab 




V 


Certified x 


Win 


Lin 


Win 


Lin 


Mac 


Win 


Lin 


Mac 


1E-5 


4.05285E+09 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


1E-6 


4.05285E+11 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


6.0 


IE- 12 


4.05285E+23 


1.0 


1.0 


1.0 


1.0 


1.0 


6.0 


6.0 


6.0 


1E-13 


4.05285E+25 


3.0 


3.0 


3.0 


3.0 


3.0 


3.0 


3.0 


3.0 


1E-100 


4.05285E+199 






Inf 


Inf 


Inf 












3.4. Results on Decisions based on Matrices. The command 
det is used by all three platforms under assessment to compute de- 
terminants. As proposed Section [2j the evaluation is based on com- 
paring the results with the certified value zero, rather than on the 
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TABLE 1 1 . LRE of linear regression results 



Matlab Octave Scilab 



Data Windows Linux Windows Linux Mac OS Windows Linux Mac OS 



RSD RSD RSD RSD RSD RSD RSD RSD 



Filip 


7.1 


8.2 


7.1 


8.2 


1.1 





1.1 





1.1 

















Longley 












































Norris 


13.4 


14.1 


13.4 


14.1 


1.8 


12.1 


1.8 


12.3 


1.8 7.9 


1.3 


8 


1.3 


10.0 


1.6 


Nointl 











Inf 


2.8 


Inf 


2.8 


Inf 


2.9 

















Noint2 











Inf 


2.3 


Inf 


2.3 


Inf 


2.3 

















Pontius 


3.6 


13.2 


3.6 


13.2 6.4 


1.6 


7.1 


1.6 


6.2 


1.6 8.0 


1.4 ! 


i.O 


1.4 


5.5 


1.4 


Wamplerl 


9.3 


9.4 


9.3 


9.4 7.0 


4.8 


6.9 


4.6 


6.6 


4.4 4.0 


3.5 3.1 


3.5 


2.7 


4.1 


Wampler2 


Inf 


14.2 


Inf 


14.2 9.6 


9.3 


9.8 


9.5 


9.8 


6.9 


8.5 6.2 


8.5 


6.0 


8.3 


Wampler3 





14.2 





14.2 6.9 





7.0 





6.6 


2.9 


3.5 





2.7 





Wampler4 





1.6 





1.6 6.9 





7.0 





6.6 


2.9 


3.5 





2.7 





Wampler5 





1.6 





1.6 6.5 





7.0 





6.2 


2.9 


3.5 





2.7 






numerical value itself. This is due to the fact that more often than 
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not what users are interested upon is a decision, and not a numerical 
value. 

Curiosily, the number of correct results of comparing \M\ with 
zero was the same, that is, exactly 146 for the three platforms under 
assessment. 

The results of computing spectral graph analyses are presented in 
Table M 

4. Conclusions 

Regarding the computation of basic statistics, Table [1] shows that 
the mean poses little difficulty for the platforms, with the exception 
of Octave for Linux, which presented the smallest number of LRE 
in five of the nine datasets (LRE(x, c) < 7). Surprisingly, these five 
datasets offer low numerical difficulty 

When computing the standard deviation, Octave presented the 
bests results when comparing with others two platforms. The version 
tested here was better than the one tested before (see [8]). Scilab 
presented an unacceptable low accuracy in a single dataset, for which 
LRE(x, c) < 5. 

As in other studies, c.f. reference [2], the first-lag sample auto- 
correlation is a challenging quantity to compute. None of the plat- 
forms here tested provided acceptable results. All of them computed 
LRE(x, c) < 5, and Scilab had the worst performance. 

Scilab is also the worst platform with respect to the stability of 
the results, as measured by estimated standard variation of the ob- 
served LRE. As can be noted in Table HJ all conclusions about the 
standard deviation may be reverted with small perturbations of the 
original input, e.g., the best results which were produced for the Lew 
dataset can be turned into unacceptable by subtracting slre from 
the observed LRE. 

Two other cases are notorious for their instability: the sample 
mean and the sample standard deviation, both computed by Octave 
under Linux. In most of the other cases, small perturbations of the 
original input do not change the conclusion about the precision. 



TABLE 12. Accuracy computing spectral graph analyses 





Windows 






Linux 










TYl) TL t\ £ m -\- n £g £ n £ m 


t N 






Im 






9,10 15.2 
99, 100 12.7 
999, 1000 10.5 


Inf 15.8 14.9 14.9 
14.8 Inf 14.6 13.8 
14.5 15.1 13.8 12.6 


11.1 
7.1 
1.3 


62.5 15.2 
4.1 12.7 
2.3 10.5 


Inf 15.8 14.9 14.9 11.1 
14.8 Inf 14.6 13.8 7.1 
14.5 15.1 13.8 12.6 1.3 


62.5 
4.1 
2.3 






2,17 15.3 
2, 197 14.3 
2, 1997 13.1 


Inf 15.7 Inf 14.7 12.1 

15.8 15.2 inf 11.7 6.1 

15.9 14.2 15.9 9.6 1.1 


100.0 15.3 
100.0 14.3 
13.1 


Inf 15.7 Inf 14.7 12.1 100.0 

15.8 15.2 inf 11.7 6.1 100.0 

15.9 14.2 15.9 9.6 1.1 








Windows 






Linux 






Mac OS 




rn,n l\ £ m +n Is ?n l-m 


In 




£n 


&m-\-n &n 




Octave 


9,10 15.4 
99, 100 13.1 
999, 1000 11.1 


15.7 15.5 15.0 14.9 37.5 

15.2 15.7 14.3 14.5 5.1 

14.3 14.9 13.1 13.3 0.8 


11.1 15.4 
13.1 13.5 
1.0 11.1 


15.7 15.5 15.0 14.9 

15.2 15.7 14.3 14.5 

14.3 14.9 13.1 13.3 


11.1 
5.0 
0.8 


37.5 15.2 
9.2 13.0 
1.0 11.1 


Inf Inf 14.8 15.1 
15.3 15.4 13.9 14.1 
14.2 14.9 12.7 12.7 


12.5 22.2 
8.0 7.1 
1.6 0.5 


2,17 14.9 
2, 197 14.4 
2, 1997 14.1 


15.4 Inf 14.5 15.2 
15.9 14.7 12.1 14.4 
15.6 Inf 15.0 9.7 







6.2 14.9 
17.8 14.4 

4.3 14.1 


15.4 Inf 14.5 15.2 
15.9 14.7 12.1 14.4 
15.6 Inf 15.0 9.7 







6.2 15.4 
17.8 14.9 

4.3 13.1 


Inf 15.6 14.3 15.3 
Inf 14.8 12.2 14.5 
15.6 14.8 9.6 14.0 


37.5 
7.6 
1.9 


Scilab 


9,10 14.6 
99, 100 13.4 
999, 1000 12.1 


Inf 15.8 15.1 14.9 37.5 
15.5 15.7 14.6 14.2 5.1 
14.9 15.6 13.6 13.3 1.1 


44.4 15.2 
10.1 14.3 
3.9 11.4 


Inf 15.8 15.4 15.0 25.0 
15.5 14.8 14.7 Inf 7.1 
14.9 15.6 13.6 14.1 1.0 


44.0 15.2 
6.1 13.0 
1.8 10.9 


Inf Inf 14.8 15.1 
15.4 15.7 13.8 13.8 
14.2 14.8 13.1 13.1 


12.5 22.2 
4.1 5.0 
1.3 1.1 


2,17 15.0 
2, 197 14.5 
2, 1997 13.0 


15.4 Inf 15.5 15.2 
15.8 14.8 14.7 12.3 
15.6 14.6 Inf 9.9 





100 


16.2 15.0 
20.9 14.5 
13.5 13.0 


15.4 Inf 15.5 15.2 
15.8 14.8 14.7 12.3 
15.6 14.6 Inf 9.9 





100 


16.2 15.5 
20.9 14.2 
13.5 13.1 


Inf 15.7 15.4 14.4 
15.6 14.3 14.0 12.0 
15.6 13.5 12.9 10.0 


37.5 
17.3 
8.0 
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Scilab presented the best performance when dealing with the bi- 
nomial and t-Student distributions, and also when computing the 
cumulative distribution function of the Poisson law (LRE(x, c) > 6). 
In this last, Octave and MatLab presented better results that their 
previous versions (see [8]). 

When computing the F distributions, Octave produced the best 
results, mainly if compared with its previous version; as presented by 
Frery et al. [8], this platform had produced the worst answers. But 
Octave fails to produce acceptable results when dealing with the 
binomial and t-Student laws. Regarding the normal distribution, 
MatLab and Octave obtained the same good results, while Scilab 
produced bad results. The three platforms were acceptable when 
dealing with the gamma law, that is, in this case LRE(x, c) > 6. 

Matlab and Octave failed at computing the t-Student distribution; 
in every assessed case, there was no match or they returned an error 
message. This is a serious issue due to the widely spread use of this 
distribution in statistical tests. 

Six out of eleven linear regression datasets were not adequately 
dealt with by any of the considered platforms. Only Matlab provided 
acceptable results for Filip, Norris, Wamplerl and for Wampler2. 
Wampler2 was acceptably treated by Octave under Windows and 
Linux and Scilab under the three operational systems. Again, no 
single platform can be advised as safe for the linear regression prob- 
lems here considered. 

Suprisingly, the same results were provided by the three platforms 
when making decisions about the determinant of ill-conditioned ma- 
trices under the three operating systems. The number of erroneus 
result was acceptable, that is only 94 in 240 logical comparisons 
with the value zero. Nevertheless, users are advised to be very care- 
ful when testing equality between a value of interest and a numerical 
computation involving determinats in these platforms. 

The assessment based on spectral graph analysis presented a very 
consistent behavior with respect to the problem size (the bigger the 
graph, the worse the answer), being £ M and £n the most sensitive 
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quantities across all platforms and operating systems, and they can 
be reported as good in most cases. The first and last eigenvalues (£i 
and t m +n) are always dependable if computed in double precision 
and then tested in single precision, being the latter consistently more 
precise than the former. The balance of bipartite connected graphs 
did not have a strong impact on the results, except for the percentage 
of correct eigenvalues. 

Extreme care must be taken when making decisions about graphs 
based on their spectral properties. As a rule of the thumb, double- 
precision computation is advised, but the comparison to known val- 
ues should be made rounding or, at most, using at most floating 
point representation. 

Regarding the variability among operating systems, MatLab and 
Octave were equivalent and more consistent than Scilab in most of 
the situations under assessment. 

The results are the same in platforms under 32 and 64 bits oper- 
ating systems, so the latter were not reported in the tables. 
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